Metrics and Mistakes

Story: Blog Tourism (aka seeking many visitors)

Some years ago I worked with a company which worked in a highly specialised area. This describes many companies I have worked for or with in the past two decades. All stories on this site are anonymised and for the purposes of illustrating a point via storytelling. Consider them fictional if you prefer. Please do not attempt to correlate this story to any specific company; (a) it’s polite not to; (b) you’ll be wrong! Our blogs and articles were usually technical: ones that helped customers learn, understand technology, understand what new features let them achieve, how-to guides, key knowledge, and so forth. Mostly solid technical content; all of it was concretely useful.

Some articles had low hit counts. In this story’s environment, these ones were valuable to our audience because they were not widely known information, hard to discover, yet critical if you happen to encounter a situation where this fact was relevant—but also, and this is key, not read very often.

These were cases of interesting or useful, high value, but rarely visited. One example might be a support article for an unusual but difficult situation.

For a while, we had a Marketing lead who believed we should only have blogs and articles that had a high number of visitors. We saw a lot of low-content, high-fluff pieces created by writers without domain expertise (but who were cheap) Today this would be AI-generated content. which were not very related to our product or technology, were inspecific and light on useful detail, but which gained visitors from generic search keywords.

These were cases of uninteresting, not relevant, low value, but often visited.

At the same time, the Marketing team was migrating the blog/article engine, and were migrating blogs by hand, via overseas workers contracted to copy-paste text and manually migrate from one database to another. Since this was slow, error-prone and expensive, to save costs a large number of articles needed to be culled, and the metric by which this was chosen was the number of visitors to each post.

The low value but new often-visited articles made the cut; the less-visited but high-value articles did not. My team fought strongly for interesting or even historically valuable content to be preserved in the migration, and much of it indeed was preserved as a result. Sadly I cannot find the specific articles I was thinking of when writing this on that company’s site today.

Here, I’ve spoken of ‘high value’ being interesting technical content, and I’d like to skip past whether that is a good judgement of value or not and for this article, accept that yes, it is a good basis for value. What I’d like to focus on is the metric chosen to represent value to the business for each blog post, and that was hit count: number of visitors.

What happened once that metric was chosen and applied?

Judgements and actions were taken based on that metric.

And that meant that low-value and low-content blogs that gained lots of visitors were kept, high-value domain-relevant blogs were deleted.

Now, hit count is not necessarily a bad metric for value. However: when we saw the change in content style driven by the marketing team, and their happy announcements of high hit rates, I asked questions about the behaviour of the visitors who landed on those blogs. Did they stay on the page long? Did they move on to other areas of our website? Did they download a trial? Could we convert them into leads? Did we have any indication from their tracking profiles that they were in our market segment, or students? Was there any indication at all that these were potential customers?

The answer was ‘no’ to all of those. If business value is building a community around your product, or driving leads for sales, these blogs failed on both counts. But that didn’t matter. The team had chosen a metric and by that metric were doing very well.

◆     ◆     ◆

Polish-American philosopher Alfred Korzybski said ‘the map is not the territory.’ In this analogy, metrics are the map, where a map is not real: it’s only what a surveyor saw and illustrated. We’ll return to this.

Metrics as Proxies

A metric is a proxy. A stand-in, something meant to represent something else. This is forgotten by many people. When that marketing team chose a metric of high blog hit counts, they were originally trying to ensure we had blog content that was interesting to a large number of our product audience (and therefore sales and therefore revenue) audience. The proxy for being interesting to our audience was how many people visited the blog posts.

There was no actual value in having many people visit the blogs if they were not interested in the blog content, did not stay on the site, and were not someone who might ultimately purchase the software the content is created for.

This means that by itself, hit count was a poor proxy for measuring the actual goal (growing our-market-segment audience.) In a second story I haven’t told here, but which you can guess, Facebook growth and interaction growth was similarly a poor proxy for that goal (growing visibility among our market segment and potential customers on social media.)

When you measure something, if you have no way to directly measure what you’re interested in, you’re forced to measure something else: to choose a measurable proxy that you believe has high correlation to your actual goal or interest. Make sure you choose your proxies well.

Acting on Metrics

Because you are almost always measuring proxies for what you actually want to measure, you can only behave as though the metric is measuring the proxy, not as though the metric is measuring what you want to measure. This is forgotten by many people. Once you see data from your metrics, you will take action based on those results, action based on the metrics. This is reasonable: if your goal is gaining views among your audience and you see that certain types of blog posts attract more of your audience, you’ll try to do more of what was successful. Note that this itself is a proxy: what certainty do you have that market-audience visibility correlates to your actual goal of making more sales of your tool? (It’s likely, sure. But how are you sure? Is this worth questioning?) Sometimes it seems all marketing and outreach behaviour is a proxy for something else, a deep rabbit hole. Sanity comes not from recursively following this but by returning to: choose your proxies well. Make them as direct as possible. If your metric directly measures your goal, this is fine, but because almost always your metric measures a proxy for your goal, once you take action based on the metric you are not taking action based on your goal but on something other than your goal.

This marketing team started optimising marketing content based on the hit count metric, and it worked: lots of hit counts. But it had no value for the goal that the hit count metric was a poor proxy for: content interesting to people in our market and encouraging use of the product we wanted to sell. This meant that the marketing team was being ineffective and less successful at their own goals: reaching out to the potential sales / revenue audience.

But even though they failed their purpose, they had great results for their metrics.

Being circular, if a team having great results on their metrics is the metric by which the team’s success is judged, that itself can be a poor choice of proxy for the team’s actual success at their goal. It’s a circular, recursive rabbit hole! Now consider if this can apply elsewhere: interaction analytics? Team KPIs?

Luckily, in our case, we had management that was insightful enough to view that team’s results skeptically, and we changed to different people and a different approach. Not all companies or departments have management wise enough to question the choice of metrics and to question apparent success.

◆     ◆     ◆

Let’s return to Korzybski’s map and territory, this time via poetry: Wallace Stevens wrote in ‘Not Ideas About the Thing But the Thing Itself’, which I love for the title alone: Not Ideas About the Thing But the Thing Itself. How well this captures the essence of insight! Or, an easy mistake in thinking—in general, not just metrics. ‘It was like / A new knowledge of reality…’.

Switching from viewing a metric not as a measurement, but as a measurement of a proxy, is key. Through your metrics you are trying to understand reality: the switch to understanding metrics as proxies is a new knowledge, or understanding, of reality.

And:

Say it, no ideas but in things—
nothing but the blank faces of the houses
and cylindrical trees
bent, forked by preconception and accident…
William Carlos Williams

Williams was reacting against symbolism and abstraction in poetry and trying to focus on ‘the thing’ itself. See A Place For Abstraction. The philosophical reaction Williams advocated in writing poetry is a guide for us moving away from the abstraction of metrics as a symbol or standin, to being aware of, or focus on, ‘the thing’ they are trying to measure. That article, by the way, ends up pointing out the value of abstraction in poetry. ‘Forked by preconception and accident’: does this match your use of metrics?

◆     ◆     ◆

Metrics do not measure your goals. Metrics measure a proxy. The proxy may or may not be aligned so that the measurements return close to what you would see if you could measure your goals themselves.

Never act to optimise a metric. Only act to optimise what the metric is a proxy for: your actual goal. When the metric becomes the goal, you’ve made a mistake. If you act on metrics, ensure you act on your goal.

Many people forget this.

◆     ◆     ◆

There are three takeaways:

1 — Metrics are proxies;

2 — Choose your proxies wisely;

3 — Act on the goal not the metric.